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ABSTRACT 


Effective decision making in aeronautical environments, which often involves high 
elements of risk, is critical to mission success. Unfortunately, no proven methodology 
exists to train pilots to make successful decisions. Cockpit decision making has relied on 
traditional analytical models and methodologies that underestimate the role of pilot 
experience, expertise and judgment. Naturalistic Decision Making models (NDM) 
contend that decision makers facing real-world decisions use experience and judgment to 
make timely decisions without analyzing a multitude of alternatives. 

This thesis analyzes 438 P-3 aviation hazard reports (hazreps) to ascertain which 
cognitive strategies from either the analytical or naturalistic methodology are more 
appropriate for handling malfunction situations. The author presents a hybrid model of 
decision making by P-3 pilots based on the results of the analysis and strategies from both 
methodologies. 

This thesis recommends that decision making training be treated as a core activity 
of pilots not only in flight school, but after qualification is complete. Training pilots to 
become experts will improve situational awareness and reduce the number of unfavorable 


outcomes in hazardous situations. 
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I. INTRODUCTION 


A. BACKGROUND 

Aviation is a rich domain for studying decision making. Pilots frequently face 
situations in which the consequence of an incorrect decision may be the loss of life or 
expensive equipment. These decisions often are made in situations that present 
ambiguous information, impose severe time constraints and contain exceedingly high 
levels of risk. Given the costs and high levels of risk, it stands to reason that the military 
would have conducted thorough studies of cockpit decision making and developed a 
rigorous approach to training decision makers. However, this is not the case. In fact, 
pilot decision making skills often are considered a result of such nebulous concepts as 


"professionalism," "good headwork" or merely the "right stuff." In the absence of 


empirical knowledge regarding the best way to train decision making skills, standard 
analytical models of decision making simply have been assumed to be the most 
appropriate methods (Stokes, Kemper, & Kite, 1997). For example, Besco et al. (1994) 


asserted: 


Decision making in the cockpit follows traditional views of decision 
making...in which the decision maker is: 1) presented with a situation that 
requires a decision; 2) the nature of the situation is assessed by the 
decision maker; who 3) determines the availability of alternative outcomes 
to respond to the situation; and 4) after evaluating the risks and benefits of 
each alternative; 5) selects an alternative in response to the needs of the 
_ Situation. (Besco, Maurino, Potter, Strauch, Stone & Wiener, 1994, p. 43) 


Although the analytical method of decision making is clearly valid under some 
conditions, it does not take into account the experience, judgment and expertise of the 
pilot. The field of Naturalistic Decision Making contends that decision makers faced with 
real-world decisions use experience and judgment to make timely decisions in actual 
settings without analyzing a multitude of alternatives; indeed, they frequently consider | 
only a single, workable course of action. ~ 

While decision training is incorporated into the Navy's Aircrew Coordination 
Training course, it is based solely on the analytical methodology. To the author's 


knowledge, it does not recognize many of the cognitive strategies used in Naturalistic 


Decision Making that could benefit pilots faced with ambiguous, unstructured situations. 


B. OBJECTIVES 


This thesis examines decision making in P-3 flight stations involving situations 
that meet the criteria for submission of a Naval Aviation Safety Program hazard report. 
The thesis objectives are to: (1) ascertain what types of decisions are made in different 
situations; (2) determine which decision making methodology, Analytical or Naturalistic, 
is most appropriate for a given situation; and, (3) identify the leading cause for 
unfavorable outcomes for those hazreps resulting in an unfavorable conclusion. 


Accomplishing these objectives will provide the requirements necessary to better train 


and equip decision makers in an aeronautical environment. 














C. THESIS OVERVIEW 

Chapter II discusses the process of human decision making and the need for 
gaining situational awareness. It reviews the literature to present both Analytical and 
Naturalistic Decision Making perspectives and the strengths and weaknesses of each. 
Several models of decision making, situation assessment and cognitive appraisal are 
presented as background material. me 

_ Chapter II outlines the methodology used for analyzing the hazard reports for 
this study. A brief history of the Naval Aviation Safety Program is presented to give the 
reader insight into the submission procedures and content of a hazrep. The author's 
expert credentials are reviewed to lend credibility to the analysis. The categorization 
schemes of the hazrep are identified and explained. 

Chapter IV presents the results of the analysis and discusses the applicability and 
relevance of the situations encountered to the two decision making methodologies. 
Explanations for both favorable and unfavorable outcomes are given and discussed. 
Finally, a hybrid model of decision making is presented. 

Chapter V summarizes the findings and proposes recommendations based on the 


results. The recommendations are intended to both increase situational awareness among 


pilots and reduce the occurrence of erroneous indications from faulty aircraft systems. 
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Ul. HUMAN DECISION MAKING 


A. INTRODUCTION 

Decision making is a process that includes not only the decision and its 
subsequent action, but also the initial cues indicating that a decision must be made and 
assessments of the environment. After stimuli have occurred, the-decision maker must 
accurately assess the surroundings and develop situational awareness to make an effective 
decision. This chapter discusses the processes involved in assessing a situation and the 
use of different cognitive strategies to make a decision. A representative sampling of 
models is presented as background material for their subsequent synthesis into an 
aeronautical decision making model. 

The literature on decision making indicates that there are two distinct methods of 
decision making: analytical and intuitive. Analytical methods are based on a structured, 
logical analysis of the situation, while intuitive decisions are based on pattern recognition 
and experience. Intuitive decision making is commonly referred to as Naturalistic 
Decision Making (NDM), which has its roots in research on how decisions are made in 
actual, non-laboratory settings by experienced people. It asserts that people make 
decisions based on recognizing a situation as similar to situations stored in memory and 
applying responses that worked well to these situations in the past (Klein, 1986). 
Analytical decision making relies on reasoning, precise information and timely 


™~ . OL 


calculations (Wright, 1974) while NDM relies on experience and judgment. 


B. OODA LOOP 


When discussing decision making in any context, it is often helpful to model the 
decision making process. The figure below illustrates the OODA loop as modeled by 
Colonel John R. Boyd in A Discourse on Winning and Losing, which is discussed in 
Naval Doctrine Publication Six, Naval Command and Control (1995). While normally 
thought of as being applicable to a conflict between parties, this model can be used to 
describe the way in which decision makers gather information to reduce uncertainty, gain 
situational awareness and make decisions regarding a course of action in any situation. 
The OODA loop is an iterative process, in which the decision maker is constantly 
engaged in collecting information, making evaluations and decisions, and observing the 
consequences of these decisions. The acronym stands for observe, orient, decide and act, 
which describes the sequence of events performed by a decision maker. A simple 


scenario involving a minor aircraft malfunction is used to help illustrate the process. 


> Observe 





Act 


Decide 





Figure 2-1 OODA Loop (adapted from Naval Command and Control) 








1. Observe 


The first step, to observe, involves gathering information on the situation from 
sources within, or external to, the decision maker's situation environment. This 
observation may occur as a result of a stimulus alerting the decision maker that an action 
is required. For example, the initial data may be a flashing oil pressure indicator light or a 


rise of oil pressure noted from a gauge. 


2. Orient 


The second step, to orient, is achieved by gathering more data on the situation 
and converting it to useable information, knowledge and understanding. The nature of 
the malfunction, seriousness of the indications, timeliness of the actions required, and 
procedures required are vital pieces of information the decision maker needs to assess the 
situation and form a mental image. 


3. Decide 


The third step in the OODA loop is making a decision, which is choosing a course 
of action that is either a response to the observations or a plan for future events. For 
example, a Plane Commander decides to abort the mission and divert to an alternate 
airfield after analyzing the situation and performing the applicable procedures for a rise in 
oil pressure. 


4. Act 


Taking action is the fourth step, and its execution is monitored by the decision 


maker and the pertinent crew members. In the case of an aircraft experiencing a 








malfunction, this monitoring involves both internal cues and external ieuaiie of the 
environment (in the example above, diverting to an — airfield would include 
communicating with FAA, ATC, airport personnel, etc.). A feedback loop exists to 
ensure the action taken is appropriate, so that adjustments can be made to alter the 
decision, as more information is gathered. The OODA loop is a cyclical process that 
requires feedback to function effectively. _ 

It is important to understand that many of these OODA loops occur 
simultaneously. While a Plane Commander is gathering information and observing the 
situation, the Tactical Coordinator also is assessing the impact of the malfunction and the 
impact that the resulting decision will have on the mission. At the same time, the copilot 
and flight engineers are gathering information with the intention of either making their 
own decision or providing input to the Plane Commander. An action initiated by any of 


these crew members may affect the observations of the Plane Commander and therefore 


alter his OODA loop accordingly. 


C. SITUATIONAL AWARENESS 


As can be seen in the OODA loop, it is necessary to observe and orient prior to 
deciding on a course of action. During this phase the decision maker conceptualizes the 


environment, or attempts to gain situational awareness. Situational awareness is formally 


defined as: 


™~ . 


the perception of the elements in the environment within a volume of 


time and space, the comprehension of their meaning and the projection of 
their status in the near future (Endsley, 1988, p. 36). 

















Endsley states that decision makers often have an internal representation or a 
mental model comparable to the situation with which they are dealing. This mental model 
is compared to the current situation and projections are made (based on the degree of 
match) to predict events as they should occur in the future. This is the basis for two of 
the situational awareness models discussed in this chapter. A more specific definition of 
situational awareness, borrowed from a U.S. Navy Aircrew Coordination Training (ACT) 
Course, is: "the extent to which a pilot's perception of his environment mirrors reality." 
(Patrol Squadron THIRTY ACT Course, 1991) 

Assessing and correctly understanding the situation frequently poses the major 
portion of any decision making task related to a malfunction or emergency condition in an 
aircraft. In almost any situation, effective decision making depends largely on having a 
good understanding of the situation and one's surroundings. Too often, situation 
assessment is not given due diligence, and situational awareness is either never developed 
or lost. In fact, many human errors attributed to poor decision making can be traced to 
errors in correctly assessing the situation and, as a result, taking inappropriate action 
based on this initial erroneous situation assessment. An example of incorrectly assessing 
a situation is an aircrew who secures a perfectly good engine as a result of an erroneous 
turbine inlet temperature indication, believing an overly high temperature condition exists. 
While securing the engine would have been the correct procedure if an actual overly high 
temperature existed, in this case it is unnecessarily shut down and salon the crew ina 


potentially hazardous situation. 








The following discussions explain the underlying processes and mechanisms in 
developing situational awareness and the relationship between this assessment of the 
environment and the decision making process. 

1. Cognitive Hierarchy 

Numerous Service Publications (e.g. NDP 6, Naval Command and Control, 
Marine Corps Doctrine Publication 6, Command and Control) refer to 'a model of the 
Cognitive Hierarchy to characterize how a decision maker deciphers information to 
achieve full understanding of a situation. Figure 2-2 (adapted from NDP 6, Naval 
Command and Control) illustrates the four steps in generating this understanding, or 
situational awareness, and can be viewed as the process used during the first two steps of 


the OODA loop, observe and orient. 







Judgement 


Knowledge 


= 


Information 


Figure 2-2 Cognitive Hierarchy (adapted from Naval Command and Control) 
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Data is meaningless to a decision maker without processing. For instance, 
electronic signals generated by a tachometer generator on the reduction gear box are 
simply not useable until they are processed and transformed into a recognizable and 
meaningful format or display. Once this processing occurs, the data, now integrated in 
the form of information, is placed in its situational context (RPM readout on a flight 
station gauge). Using cognition wil learned aspects of the implications of RPM and 
propellers, the decision maker can integrate various pieces of information to build an 
informed picture of the situation. Perhaps an audible pitch change accompanies hee 
appears to be an RPM over speed condition on the gauge. This analyzing and fusing of 
various pieces of information serves to both validate the information as factual and build 
a clearer picture of the situation. Finally, judgment is applied by the pilot to transform 
the knowledge into understanding. Intuition, experience, expertise and training are all 
relevant to judgment and understanding. The consequences of allowing an over speed 
condition to exist on an operating engine and the ramifications of acting on the condition 
are now Clear to the decision maker as understanding emerges from knowledge. 

While it is certainly desirable to achieve full understanding or perfect situational 
awareness of a situation before making a decision, it is obvious that many decisions must 
be made in an environment of ambiguous, erroneous, missing, or conflicting information. 
To make matters worse, many decisions must be made in an environment constrained by 
time-pressure and considerable amounts of stress. As uncertainty, ambiguity and stress 


increase; the pilot must find other methods to synthesize the available cues into an 


il 





accurate understanding. Experience may be the key to understanding and grasping the 
situation faster and more reliably. 


2. Cue Recognition 


It has been hypothesized that experienced pilots differ from inexperienced pilots in 
the ability to retrieve domain-specific knowledge representations from long-term memory 
(Stokes, Belger & Zhang, 1990). Thus, when confronted with -situations that could 
endanger the safety or efficiency of the flight, experienced pilots may more readily 
recognize cues relevant to the problem. They compare these cues, through a process 
referred to as "pattern matching,” with situational schemas stored in long-term memory. 
Only if they are unable to match cues to their mental maps do pilots drop into an 
alternative strategy using real-time computational and inferential processes of working 
memory (Stokes, Kemper, & Kite, 1997). Inherent in working memory is a "thinking 
balloon" of: cognitive appraisal that may expand in size and detract from working 
memory's capacity as time-pressure, stress and uncertainty increase. Figure 2-3, adapted 
from PC Based Instrument Flight Simulation (Sadlowe, 1991), illustrates the process of 
pattern matching in long-term memory versus the analytical processing done in working 
memory. Most of a person's active processing of information must occur in working 
memory. New information must be combined with existing knowledge and a composite 
picture of the situation developed for understanding to occur (refer to Cognitive 
Hierarchy). Projection of future events and eheauik decisions regarding appropriate 


™ so 


courses of action must occur in working memory as well. For inexperienced decision 


makers, or those dealing with unique situations, working memory may constitute the 
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Figure 2-3 Pattern Matching Model (adapted from PC Based Instrument Flight Simulation) 


main bottleneck for situational awareness. Within this model, experienced pilots exhibit 
more effective and efficient decision making as they can more readily identify significant 
cues and gain situational awareness than can the pilot who must integrate cues and 
generate an original solution. 


3. Noble: Situation Assessment 


Noble's model of situation assessment is similar to the Cue Recognition model 
above. Information on previous experiences is stored in the decision maker's memory in 
order to form an interpretation of the current situation (Noble, 1989). The process of 
performing . situation assessment unfolds as follows (Figure 2-4): first, concrete 


information about the situation is combined with environmental, or "context" information. 
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Then, this current situation assessment is compared with knowledge retrieved from the 
decision. maker's long-term memory to form a tentative interpretation (representation) of 
the situation. This representation includes certain expectations concerning future 
concrete and contextual information. These expectations are tested by this additional 
information from numerous sources. To the extent that the expectations do not match 
this information, the representation is refined or rejected in favor of a new representation 
that is tested, retained, refined or rejected in turn. Thus, people sometimes assess and 
decide by observing how the current situation is similar to previously observed situations 


and implementing actions that worked in these previous situations. 
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Figure 2-4 Noble's Situation Assessment Model (1993) 
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D. AN ALYTICAL DECISION MAKING 
1. Background 


Analytical decision making can provide excellent results and is well suited for 
problems that have an accurate information base with clearly defined goals. It is soundly 
based in logic and is specifically designed to eliminate biases and intangible or qualitative 
factors that are difficult (if not impossible) to measure. The process can be reduced to 
algorithms and programmed in a computer. These processes can be easily taught and are 
independent of the knowledge and experience of the decision maker. This implies that a 
junior, inexperienced decision maker will reach the same answer as an expert if they have 
the same data and use the same process. Analytical decision methods also tend to give 
the decision maker a clear, defined rationale to justify and document their decisions, 
especially when rationalizing an optimal solution with clear goals. 

Analytical decision making may be best used for predeployment and preflight 
preparation planning rather than "heat of the moment" or emergency scenarios. 
Administrative and logistical planning scenarios may also lend themselves to analytical 
methods to best optimize resources. 

The classical model of analytical decision making holds that it is a rational and 
systematic process of analysis based on the concurrent comparison of multiple options. 
Bernoulli's Maximization of Expected Value is designed to choose the best option given 
ordinal or better information. For each option the product sum is computed across | 
outcomes and the option with the largest product sum (i.e., the highest expected value) is 


selected (Sommer, 1954). It usually requires some type of computational aid, demands a 
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well structured problem and often takes extensive time for application. Preflight planning 
for a transoceanic evolution is an ideal opportunity to employ an expected value 
methodology, as it requires selection of an optimal flight path given a multitude of 
resources and constraints (fuel, weather, field availability, winds aloft, etc.). 

Using this methodology is highly time consuming in both the computational 
aspects and the option generation process. For the procedure to~perform to its best 
capability, an exhaustive option production process must first be conducted to ensure 
completeness and optimization among competing alternatives. It takes a while to 
identify, analyze, and compare all the various alternatives available. As a result, no 
matter how quickly the computations can be performed, there will always be a certain 
minimum amount of time associated with a decision reached by this method. 

The analytical model also requires a high level of certainty and accuracy of | 
information.” It assumes that the information is available and reliable or, if it 1s not 
available, that the decision maker will have the ability to obtain it. This may impact the 
amount of time required to make the decision as it will certainly take time to gather more 
information. if information is missing, altered, or inaccurate, the resulting quality of the 
decision will suffer. 

Experience, perspective and judgment are not requisites of the analytical model 
and are not taken into consideration when computing the optimal solution. The aceiviical 
model is process-based and arrives at the correct solution as long as the right information 
is used in ‘aii formulation. This model's failure to consider ania of the decision 


maker results in the prediction that near-novice pilots using analytic decision making 
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methods can operate under the same conditions as their high-time seasoned peers. The 
veteran's years of experience in the flight station add no particular insight to picking the 
optimal flight path with regard to winds aloft or weather avoidance. 

The appeal of the analytical process is clearly the highly defined, clear cut, orderly 
process that it offers. Decisions made with this method have ample documentation to 
justify and rationalize the final solution. Decision makers have at their disposal a history 
of information that can be scrutinized and investigated to ensure the proper decision was 
indeed made. 

In general, analytical decision making is more appropriate under the following 
conditions: 

-Time is not a factor. 

-A choice among clearly defined options exists. 

-Complete and reliable information is available. 

-Decision makers lack experience. 

-Decisions must be justified. 


2. Disadvantages 


The biggest disadvantages of using analytical decision making in an aeronautical 
environment are the requirements for: (1) sufficient time to gather, analyze and choose.a 
decision, and (2) accurate information as an input to the process. It is often time 
consuming to compile and analyze the vast amounts of data required for the analytic 
decision aera process. Often, the information required to make ‘ie decipil is lacking, 


— 


misleading or wrong. Many situations encountered in an aircraft are unique to the 
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particular event, so the information required for the option generation phase is completely 
unknown. Further, to improve the certainty of the optimal solution, decision maker’s 
must gather even more information, which can increase the time factor considerably. One 
study also suggests that analytical decisions are shown to be less successful than other 
methods when made under time-pressure and stress (Wright, 1974). Since the analytical 
strategy can be reduced to algorithms and decision criteria, the use of analytical decision 
making 1n a tactical scenario may also give the enemy an advantage, if they were to 


process the same information and predict the decision maker’s actions. 


E. NATURALISTIC DECISION MAKING 
1. Background 
The naturalistic decision making (NDM) model is derived from the study of how 


experienced people actually make decisions in real-world situations or in "naturalistic" 
scenarios. A short definition of NDM is: “the way people use their experience to make 
decisions in field settings." (Zsambok, 1997, p. 4). The naturalistic decision making 
- movement's center-of-gravity is based on the evidence that evolved from research on fire 
ground commanders (Klein, Calderwood, Clinton-Cirocco, 1986). Klein et al.'s body of 
research essentially states that experts use their experience to adopt a successful course 
of action without applying rigorous analytical strategies or deliberating over more than 
one option prior to making a decision. Various studies of how Naval command and 
control officers and infantry soldiers decide have generated the same conclusions: "under 


operational conditions, decision makers rarely use analytical methods, and non-analytical 
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methods can be identified that are flexible, efficient ta effective.” (Klein & Klinger, 
1991). Naturalistic decision making strategies are not suitable in every situation and the 
conditions where they are most applicable include dynamic and continually changing 
conditions, real time reactions to these changes, ill defined tasks, time pressure, 
significant personal consequences for mistakes, and experienced decision makers (Klein, 
1993). = 

Using this intuitive process, experienced Recon makers assess the situation and 
try to recognize familiar patterns. The first solution, although not necessarily optimal, is 
generally workable and evaluated to be "good enough". Mental simulation, rather than 
decision analysis, expected value, or multi-attribute utility functions, is used to evaluate 
the options. The primary methods used by decision makers employing this naturalistic 
strategy are pattern matching and mental simulation. Pattern matching involves a 
comparison of the current situation with previously experienced situations. In complex 
situations, mental simulation involves applying the response that the decision maker has 
decided to use to see how the results would "play out" if a specific response is taken. 

The most significant advantages of using intuition for decision making are speed, 
creativity, tolerance of uncertainty and the ability to visualize the problem and its 
solution. In many malfunction-oriented aeronautical decisions, time is the critical factor. 
Intuitive decisions tend to be much faster as they rely on selection of a "satisficing" 
course of action (i.e., not necessarily optimal, but good enough) based on experience, 


~ 


rather than the generation and comparison of a multitude of possible solutions until the 


optimal one is chosen. 
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2. Klein's Recognition-Primed Decision Model (RPD) 

Klein's model holds that experts, who are pressed for time, while facing a task 
loaded with uncertainty, first assess whether the situation strikes them as familiar or 
unique. This process is referred to as “recognition-primed" because recognizing the 
situation as similar to some situation previously experienced is key to how the 
accompanying response is generated and the ensuing action is carried out. Typical 
situations lead to typical (procedural) actions, while novel, or unique, situations pose new 
challenges, which cannot be solved simply by applying the same routine. To recognize 
the situation and guide the selection of proper action, the decision maker identifies critical 
cues that may explain the situation and causal factors that explain what is occurring and 
what may occur in the future. Experts use prior knowledge and experience to recognize 
and classify a situation. Rather than comparing in detail the pros and cons of the different 
outcomes, they quickly imagine (via mental simulation) how one course of action may 
play out and then either choose to adopt it, if it satisfies the situation, or adapt the 
response. Experts take the first workable solution they can find, and while it may not be 
optimal, it is often quite good. 

When the situation is not recognized as familiar or typical, the experienced 
decision maker can still act. They can modify a course of action, retrieved from memory, 
that worked in a similar situation to accommodate the current situation. 

Klein ee that RPD is not a universal — of decision making. It is a 


model that is more suitable for situations that are: (1) sigiarienaed by high levels of 


siiaabtiabs and (2) likely to be encountered under time pressure by people with high 
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levels of expertise. It is less likely to be applicable if these conditions are not met, if the 
decisions are naturally presented as choices, and if the decision maker feels a need to 
optimize the decision (Lipshitz, 1993). 

3. Montgomery's Search for Dominance Structure 

Montgomery is interested in how decisions are actually made when several 
alternatives are available. His answer is simply that a decision maker will search for a 
dominant alternatrve (Montgomery, 1989). An alternative is said to be dominant if it is at 
least as attractive as its competitors on all relevant attributes, and exceeds each of them 
on at least one attribute. Essentially, Montgomery thinks of decision making as the 
process of finding a good argument for acting in a certain way, first by a quick selection 
of a promising alternative and then by testing or ensuring the dominance of this 
alternative. He feels this is compatible with the limited capacity of human information 
processing, -by focusing on a limited number of alternatives and accentuating the 
differences between them. Also, the availability of a dominant alternative helps decision 
makers to persist in its implementation and to rationalize and justify their choice. A 
drawback of this persistence in implementation may be the distortion of reality to 
maintain dominance of an alternative that is actually dominated by a better choice when 
the environment changes (Montgomery, 1993). 


4. Rasmussen's Decision Processes 


Rasmussen is interested in the decision making processes of human operators of 


complex systems. He has distinguished between three types of ‘behavior that are 
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controlled by qualitatively different cognitive mechanisms: skill-based behavior, rule- 
based behavior, and knowledge-based behavior (Rasmussen, 1983). 

Skill-based behavior includes expert sensory-motor performance, which runs 
smoothly and efficiently without conscious attention. Skill-based behavior is controlled 
by a dynamic mental model that depicts the decision maker's movements and environment 
in real time, thereby enabling him to adjust rapidly to feedback. An example of this type 
of behavior is the pilot who subconsciously applies slight back pressure to the yoke to 
prevent a loss in altitude as a minor nose down pitch occurs in the aircraft. This 
correction is automatic and taxes neither long term nor working memory. 

Rule-based behavior is controlled by rules and procedures that can be stated 
explicitly by the decision maker. This is a learned behavior and can become skill-based if 
practiced and repeated often enough. Information at this level is easily processed and 
indicates that a situation exists that requires invoking a known rule or specific action. 
Noble's situation assessment and Klein's RPD depict rule-based behavior in the early 
stages of situation recognition (Lipshitz, 1993). Pulling the emergency shut down handle 
for an indication of an engine fire is exhibiting rule-based behavior. 

Knowledge-based behavior requires a deeper understanding of the situation 
before any action may be taken. An explicit consideration of options and objectives may 
be accomplished to preclude inappropriate behavior for a misunderstood seaiainael 
Unique situations, or environments with uncertainty and/or ambiguity are cases where 
knowledge-based behavior is most likely encountered. In the case of the engine fire 


above, rule-based behavior may move up a level to knowledge-based if the fire is 
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accompanied by a simultaneous malfunction on another operating engine. An analysis of 
alternatives and ramifications of action would be conducted prior to automatically 
invoking a procedure. 


5. NDM Use 


Ambiguous, unstructured environments may be the best domains in which 
Naturalistic Decision Making is useful. Any situation requiring a rapid, almost 
subconscious decision may also be best suited for an intuitive process rather than a 
lengthy analytical one. Generally, intuitive decision making is better in conditions where: 

-Time is critical. 

-Decision makers are knowledgeable and experienced. 

-There is a high degree of uncertainty. 

-There is ambiguous or changing information. 

-Innovative or creative thought is required. 

The essential factor in NDM is experience. Experience is the critical element that 
allows for the situation assessment (which leads to situational awareness) that is at the 
heart of naturalistic decision making. Experience allows the decision maker to recognize 
a situation as typical, or if not typical, provide enough information from a similar 
Situation to turn knowledge into understanding. If sufficient experience exists then 
valuable time is saved by not having to integrate cues and derive an original solution. 


6. Disadvantages 


There are drawbacks to using NDM for important aeronautical decisions. 


Intuitive decisions are based upon the decision maker’s personal experiences and are 
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subject to individual biases, prejudices, level of experience, and ego. The decision will be 
based on memories that may be inaccurate, incomplete or influenced by personal 
preferences. The situations where intuitive thinking is thought to be most applicable, 
under high stress and time critical, may tend to inhibit the intuitive process. Naturalistic 
decisions may not be trusted by seniors, peers or subordinates, as there is little or no 
documentation to back them up. Finally, inexperienced decision makers are not able to 


make good intuitive decisions, as they have little, or no, experience to draw from. 


F. SUMMARY 


From Bernoulli's Expected Value computation to today's NDM movement, 
decision theorists have produced a multitude of hypotheses on how and why decisions are 
actually made. Each strategy has roots in some common logic, but each is unique in its 
attempt to explain He reasoning and cognitive strategies of the decision maker for a 
particular set of circumstances or situation. Whether an analytical or naturalistic strategy 
is used, all decision makers use a process that requires a stimulus, developing situational 
awareness, choosing a decisive course of action and implementing this action. Situational 
awareness is the most critical aspect of any decision, as the lack of it (or an incorrect 
situation assessment) may result in an inappropriate response to a misdiagnosed 
malfunction. 

It is important to realize that while cognitive strategies from the analytical and 
naturalistic methodologies may appear to be mutually exclusive, they can be used in a 


complimentary fashion during a single decision situation. To incorporate an analytical 
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optimization into a decision process, after an intuitive situational assessment has been 
performed, is a practical method to deal with a unique situation that is not time critical. 
For example, the crew that now has time to conduct an optimal alternate airfield search 
after conducting a precautionary engine shut down for an unknown vibration is 
combining the two methodologies. The experienced crew decides the vibration warrants 
securing the engine, but now has the time to afford a deliberate and-calculated choice of 
alternative landing sites. For any situation it is important to consider both models and 


what combination of the two will work best. 
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I. METHODOLOGY 
This study analyzes P-3 hazard reports (hazreps) to identify different types of 
decisions and decision making strategies used in an aeronautical environment. The 
following sections explain the purpose of a hazrep, the format and information contained 


within, and the analysis performed. 


A. HAZARD REPORTS 

1. Background 

Unfortunately, naval aviation mishaps are an inevitable result of operations 
involving flying. The Navy continues to strive for a reduced aviation mishap rate through 
numerous safety programs. The Naval Aviation Safety Program is such a program; it is 
designed to reduce aircraft accidents through careful investigation, documentation and 
analysis of hazards. The goal of the program is to identify aid eliminate hazards before 
they contribute to aviation mishaps. A hazard is defined as "a potential cause of damage 
or injury to aircraft or personnel" (OPNAVINST 3750.6Q, 1989, p.44). The Navy's 
mishap rate of 50 mishaps per 100,000 flight hours in the 1950's has dropped significantly 
to 0.7 per 100,000 flight hours in FY 1999 due mainly to this ongoing identification and 
reporting of hazards (Naval Safety Center statistics, 2000). 

The most efficient and direct detection of hazards often is accomplished by an 
individual or command with first hand knowledge of the circumstances surrounding a - 
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hazardous situation. All command safety programs have methods in place to identify 


— - 


hazards, such as reviews of operating procedures, safety surveys, analysis of equipment 


27 


failures and inflight incident reporting. Material/equipment failure, human error, 
physiological episodes, near mid-air collisions and bird strikes are examples of the 
multitude of inflight incidents where hazards can be identified and reported. 

The Aviation Safety Officer (ASO) is a designated naval aviator or naval flight 
officer who is a graduate of the Aviation Safety School. The ASO 1s a participating pilot 
or naval flight officer in the squadron and possesses additional traifiung as a specialist in 
the Naval Aviation Safety Program. For the purposes of investigating and reporting 
hazards, the ASO is the primary functionary in the process and authors the hazrep after 
receiving appropriate input from squadron members. It is the responsibility of the ASO (in 
conjunction with the Commanding Officer) to ensure that hazreps are submitted not only 
in the correct format but with full and forthnght information surrounding the hazardous 
situation. 


2. Format 

Once a hazard is identified, submission procedures found in OPNAVINST 
3750.6Q are followed to publish a hazrep, which is addressed to all pertinent commands, 
details the circumstances under which the hazard was identified and proposes actions to 
eliminate its future occurrence. To ensure the widest dissemination, the message is 
addressed to offices of the Chief of Naval Operations, Naval Safety Center and all 
activities that fly P-3's. The ASO assigns risk assessment codes to the hazard, indicating 
the probability of future occurrence and potential impact of the hazard if it does occur. 


™~ . 


The body of the message summarizes the circumstances surrounding the hazard 


and its specific identification along with a listing of pertinent aircraft, equipment and 
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weather data. The body of the hazrep attempts to explain the what, where, why and how 
of the hazard and provide enough details for others to fully comprehend the severity and 
repercussions if the hazard were to occur in the future. It discusses the crew's detailed 
analyses in order to give the reader insight into the pilot's decision making and thought 
processes. The corrective action section may propose a solution to eliminate the hazard 
(if one is readily apparent) without regard to cost or feasibility. Aithough a corrective 
action to eliminate the hazard may not always be readily identified, the hazrep still can be 
used as a valuable learning tool for pilots presented with similar situations in the future. 
Commanding Officer's comments conclude the message and provide a potential forum for 
further discussion of the severity of the hazard and the crew's performance in handling the 
situation. rai 3-1 shows a sample format of a general use hazrep. 

As mentioned above, the body of the message contains the relevant facts 
surrounding the identified hazard and the situation. The information presented to the crew 
from internal and external sources, the decisions made by the pilot and the actions 
performed are all outlined in the evidence and analysis section. This process mirrors the 
OODA loop discussed in the previous chapter: the pilot observes the environment, orients 
the information into some useable schema, decides on a course of action and then executes 
this course of action after the decision is made. 


3. Concerns About Quality and Content 


The quality of hazreps depends on the quality of the investigation into attendant 
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circumstances. The success of the hazrep program relies on the submission of complete, 


open and forthright information and opinions in hazreps. The exercise of command 
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influence to edit, modify, or in any way censor the content of reports is contrary to the 
spirit of the program and is prohibited by OPNAVINST 3750.6Q. Unfortunately, some 
Commanding Officers may believe that truly confessing all the bad with the good may 
harm the squadron's reputation or jeopardize future career opportunities; therefore some 
hazreps suffer from a lack of completeness. 

Another area of concern is that hazreps are not considered privileged information. 
Privileged information assures confidentiality of the information given; such safety 
information cannot be used as evidence for other investigations regarding legal matters, 
misconduct, blame, etc. The authority for granting an assurance of confidentiality is 
strictly limited to the investigation and reporting of mishaps, and not hazards. The 
instruction states that extreme care must be taken to prevent giving the impression that the 
use of the hazrep is for safety purposes only, as is the case with mishap investigations. 
This lack of confidentiality may prevent some crews from giving open and forthright 


information from fear of repercussions. 


B. EXPERT ANALYSIS 


Due to the inherent ambiguity and incompleteness surrounding some of the content 
in the body of the hazrep and the Commanding Officer's comments, and due to the lack of 
formal investigation that accompanies the situation, the research method requires that an 
expert on the subject conduct the analysis. Such expertise is required to decipher the 
nuances familiar to the P-3 community, fill in unwntten gaps when questions arise and 


provide an estimate of the severity of a particular malfunction. The ability to extract what 
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Message Originator 
PATRON XXXXXX//10// 


A ddressees 
CNO WASHINGTON DC//N889E// 
CMC WASHINGTON DC//A/SD// 
COMNAVSAFECEN NORFOLK VA//00/10/11/054// 
ALL ORION ACTIVITIES 


Header & subject line 
UNCLAS FOUO//N0370// 


SUBJ/THIS IS A GENERAL USE AVIATION HAZARD REPORT, Originator, Serial Number, 
Date, P-3, Bureau number, REPORT SYMBOL 3750-19.// ~ 


Reference 
REF/A/DOC/OPNAVINST 3750.6Q/-// 
Other references as applicable 


Summary 
RMKS/1. THIS REPORT CONTAINS A (Routine or Severe) HAZARD TO NAVAL 
AVIATION. ENDORSMENT (is or Not) REQUIRED. SUMMARY: Describe circumstances i 
brief statement of Who did What or What component failed When. 


Message Body 

2. DATA 

A. AIRCRAFT (1) Model, (2) Bureau number, (3) Modex and side number, (4) Reporting 
Custodian 

B. EQUIPMENT Nomenclature of aircraft parts or other parts involved (1) Model, (2) Make, 
(3) Part number, (4) Equipment code, (5) Configuration, (6) Work nit code, (7) etc. 

C. ENVIRONMENT List (1) Date, (2) Local time, (3) Local time zone, (4) Day or Night, (5) 
Location, (6) Altitude, (7) Weather, (8) Other background information 


3. CIRCUMSTANCES 

A. EVIDENCE AND ANALYSIS. Fully and clearly describe the hazard in terms of who did 
what or what component failed. 

B. CONCLUSIONS. Explain how hazard could result in damage and/or injury. 


4. CORRECTIVE ACTION. Describe corrective action taken to eliminate the hazard, or 
corrective action needed to eliminate the hazard. Or, if it is beyond the capability of the originato 
to formulate recommended corrective, so state. 


5. REMARKS. Content is at the discretion of the originator. Maintenance actions required to 
repair failed component or other postflight examination results can go here. 


6. POINT OF CONTACT. Name, rank, phone number for inquires. 


7. COMMANDING OFFICER COMMENTS. Commanding Officer's endorsement of the 
report 


~-- - Figure 3-1 Hazrep Format (adapted from OPNAVINST 3750.6Q) 
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may appear to be inconsequential information may provide valuable insights to the 
resultant decision. 

Based on previous experience, flight hours and schooling the author qualifies as an 
expert for the purposes of this study. With over 3500 flight hours in the P-3, he has 
personally witnessed many of the hazards identified in the 438 hazreps used in this 
analysis. As a former Fleet Replacement Squadron (FRS) instructor; NATOPS instructor 
pilot and instructor-under-training (TUT) instructor pilot, he is extremely well versed in the 
decision making processes of pilots | and of the common errors associated with task 
overload, loss of situational awareness and poor airmanship. He was instrumental in 
developing the Aircrew Coordination Training (ACT) course for the P-3 FRS and taught 
the training track through several iterations. He is also a graduate of the Naval Aviation 
Safety Officer course and has served in billets as Aviation Safety Officer (ASO) and Safety 
Department Head. As such, he has authored well over a hundred hazreps (and reviewed 


hundreds more) 1n his career. 


C. CONTENT ANALYSIS 


The hazrep was divided into several different levels based on type, class and 
category as shown in Table 3-1. The analysis was based primarily on the content of the 
evidence and analysis paragraph, with supplemental and corroborating evidence from the 
CO comments and environmental data sections. While the table shows the different 


categories, Figure 3-2 illustrates the process used to extract the data into useable 
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Ambiguous Indications (single COA) 
Rule 
nusual malfunction 


“= 
ompound factors (procedural) 
Simple procedural 
Improper procedures 
Skill Situational awareness 
Survival 


Knowledge 
Multiple malfunctions (procedural 
nique; no NATOPS guidance 
Table 3-1 Hazrep Type, Class and Category 









information for the decision making analysis. As can be seen, after the type, class and 
categories are established, each hazrep is subjected to analysis based on the use of 
judgment, the outcome, and an explanation for the outcome. 


1. Type 

Classification by type is accomplished by selecting the aircraft system that is most 
responsible for the submission of the hazrep. Table 3-2 shows the eight different 
classifications by type and the subsystems within each type. This classification by 
equipment type is necessary to aid in the identification of specific systems that may have a 
greater propensity toward failure, thus pinpointing an area for concentrating engineering 


solutions to prevent future hazards. This also may shed some light on the type of system 
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Engine ingle OOA 
fai Favorable 
Instrument Compound factors results? 
NMAC (procedural) 
Multiple malfunctions 
Other 
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results? 





Figure 3-2 Methodology Model 


malfunctions that are more prone to a specific kind of decision making, or systems that are 
more susceptible to errors due to their complexity, ambiguity, or lack of current guidance. 
Classification by type is a relatively straight forward process and is usually easily 
discernible from the subject line or opening statement of the summary section. Where 
multiple components from different systems are involved in the scenario (e.g. chips light 
and propeller malfunction), the system that is most responsible for the subsequent 


decisions and/or procedures is cited as the type. While numerous systems may be evident 
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in a single hazrep, one of them is usually the cause for the majority of decision making 
behavior. For example, a prop overspeed occurring on the return transit after an oil leak 
caused a mission abort, would be classified as engine, since the decision to act on the leak 


already has occurred. 


Instrument Navigation Ge 
Attitude source 


AVs 
Bird strikes eather 
Outflow valve 
Physiological episodes 
Air multiplier Auxiliary power unit (APU) 
efrigeration unit turbines 
Propeller Fails to feather 
Fails to unfeather 
Negative torque sensing system (NTS) 


Beta & NTS inop lights 





Prop pump lights 
aoe. Table 3-2 Classification by Type 


35 





2. Class 


Sorting the hazreps by class is a bit more difficult than separation by type. This 
process requires grouping the hazreps based on their environment, using the 
characteristics of decision processes identified by Rasmussen (1983): knowledge-based, 
rule-based or skill-based behavior. "Behavior" in this analysis may be appended with 
"environment," as the classification is based on the circumstances presented to the decision 
maker as well as the resulting behavior. However, this is a fine distinction as it is almost 
certain that a knowledge-based environment requires knowledge-based behavior, since the 
situation requires a deeper understanding and explicit consideration of options and 
objectives. 

The following definitions apply to each of the classes: 

Knowledge-based - requires a thorough understanding of the situation before any 
action may -be taken; multiple alternatives may be available depending on situation 
assesemica and judgment of the decision maker. 

Rule-based - rules and procedures can be stated explicitly by the decision maker; 
specific action is required for given cues and indications. 

Skill-based - evidence of expert sensory-motor performance without conscious 
attention (automatic stimulus-response reactions); skill errors. 

Note that the skill-based environment is broadened from merely subconscious 
sensory-motor actions to one that includes immediate "survival" reactions (near mid-air 


™s~ 


collision avoidance) and skill-based errors. These survival reactions are automatic 


—_ - 


stimulus-response reactions that are so well learned that they appear to be automatic. 
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Skill-based errors include those hazards that are a direct result of improper procedures 
being performed (or proper procedures not being performed) and should not have 
occurred. An example of this situation is a three engine landing for an oil leak that is 
caused by an improperly seated oil cap. The skill-based error occurs during preflight when 
the cap is improperly replaced and this is not noticed. 

The following sections develop these definitions of knowledge-based, rule-based 
and skill-based classes further as the categories within each are defined. The most difficult 
part of the classification is sischmninating between knowledge-based and rule-based when 
the situations become more complex. The main distinction is based on whether there is a 
single course of action and whether the decision maker has to use experience and 
extensive judgment in the decision making process. 


3. Categories within the Classes 


a) Knowledge-based Categories 


Table 3-3 shows the five categories within the knowledge-based class and 
the accompanying classifying criteria. The categories are: ambiguous indications, 
compound factors, multiple malfunctions, unique situations with no NATOPS guidance 
and unusual malfunctions. The hazreps fitting these categories are those involving 
ambiguous data or cues, multiple course of action scenarios where the pilot must make a 
judgment or choose among many alternatives, and unique, or unusual, situations that 


cannot be solved with a simple application of NATOPS procedures. 
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b) Rule-based Categories 


Table 3-4 shows the six categories within the rule-based class and their 
accompanying classifying criteria. The categories are essentially the same as the 
knowledge-based categories with the addition of simple procedural. 

The distinction between the knowledge-based and rule-based categories 
with the same names lies in the situation. If the situation is one that-can be handled with a 
strict set of rules or procedures, then it is rule-based, even in the event of compound or | 
multiple malfunctions. If the resulting action is the same regardless of the number of 
malfunctions or complexity of the situation, then it remains rule-based. For example, a fire 
warning that is not evidenced by an aircraft system warning but 1s seen visually by a crew 
member who notices a torch emitting from the turbine (ambiguous indications) is classified 
as rule-based, since the procedure is to secure the engine. On the other hand, a fire 
warning from the aircraft system that continues after the engine is secured, but is not 
accompanied by visual or other indications, is classified as knowledge-based, since the 


decision to release subsequent fire extinguishing agents is left to the pilot's discretion. 
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c) Skill-based Categories 


Table 3-5 shows the three categories within the skill-based class and their 
accompanying classifying criteria. The categories are: improper procedures, situational 
awareness and automatic stimulus-response. The improper procedures and situational 


awareness categories relate to errors made that are a direct cause of the hazard. The 


— 


resulting malfunction would not have occurred if the person or persons responsible for the 
error had properly performed the procedures. These were special case categories created 
to examine how skill based behaviors affect decision making and record how often they 


are responsible for hazards. 


Skill Scenarios involving: Sample CO comments 
Categories 

Improper Actions of the pilot or crew cause a malfunction to occur Never should have 
procedures e.g. high angle of attack on climbout in a heavy aircraft 
ith high power settings causes fire warning) 
Oversight or preflight errors cause malfunction (e.g. oil cap 
improperly replaced after servicing, brakes not set) 
Normal inflight procedures performed incorrectly (e.g. 
improper fuel crossfeed results in fuel starvation 
and flameout of engine, altimeter conversion) 


naware of surroundings (e.g. improper malfunction set up SA could have 
in training scenario, missed comms, gauge reading error) Prevented... 


naware of implications on system (e.g. failure to Should have queried 
Auto stimulus- [{mmediate evasive action (e.g. NMAC without time to judge 
response losure or distance, vehicle on runway) 


Table 3-5 Categories in Skill-Based Class 



























hain of events should 
Ihave been broken... 


















awareness 





4. Judgment 
The question as to whether the situation requires judgment on the part of the pilot 


is slightly subjective. If the pilot uses discretion in the application and/or timing of 


procedures or 1s presented a choice of actions, then judgment is considered to have been 
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used. If the pilot has a decision regarding a best case/worst case scenario, then judgment 
is considered to have been used. An example of this is the scenario in which an unknown 
fluid leak appears on the engine nacelle. If there are no indications on the flight station 
gauges to confirm the leak, the pilot may reason that the worst case is a fuel leak that 
presents the possibility of fire, while the best case is either condensation or an insignificant 
amount of oil. It is then up to the pilot's discretion regarding what course of action to 
take -- secure the engine and abort, continue engine operation and abort, or continue 
engine operation and the mission. Judgment is not limited solely to the knowledge 
category, as some rule-based scenarios require the use of judgment when deciding on 
aborting the mission, possible alternate airfields or the sequencing of procedures. 


5. Favorable Outcome 

A hazrep with a favorable outcome is one in which the correct decisions are made 
for a given scenario and the actions taken are correct for the malfunction. The second half 
of this rule is in place for two reasons. First, to prevent equipment malfunctions from 
going undocumented and second, to note the improper performance of procedures after 
correctly diagnosing the situation. For instance, a three engine landing for a fire warning 
is the correct decision for this malfunction. However, if the fire warning is due to a faulty 
fire warning element, then an unnecessary three engine landing has been performed. The 
crew made the right decision based on the information vioaeaiil but this is not a 
favorable outcome. Similarly, the pilot that elects to secure an engine for an unknown - 


fluid leak (assuming worst case scenario) may have made the correct decision if the fluid 


was indeed a fuel leak, but not if it turns out to be a slight oil leak that is within limits. 
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These unfavorable outcomes are not a reflection on the pilot, but rather on the aircraft 
detection and warning systems. The unfavorable outcome for the crew that correctly 
diagnoses the situation but performs the wrong procedures is obvious and needs no further 
explanation. 


6. Explanations for Outcomes 


The author's best attempt 1s made to assign an explanation te those hazreps where 
an unfavorable outcome exists, judgment has been used, or a favorable outcome is 
achieved and the cause is readily apparent. For the majority of hazreps in which a 
favorable outcome exists (especially those in the simple procedural rule-based category) 
there is no assignment of correct decision, as this is implied by the lack of an unfavorable 
assignment. Table 3-6 lists the explanations used and the definitions adhered to for their 


assignment to a hazrep. 


D. SAMPLE HAZREP ANALYSES 


Appendix A presents an analysis of three hazreps based on the rules and criteria 
outlined in this chapter. Actual sample "EVIDENCE AND ANALYSIS," "REMARKS" 


and "CO COMMENTS" paragraphs are quoted for the reader. 
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Favorable outcome Behavior exhibited / Substantiating criteria 
explanations 













orrect decision Action is appropriate for the situation; Decision does not 
— | nvolve expertise or great thought, but an incorrect choice 
is also available; lucky choice made _ 
Ex tSA Pilot has outstanding grasp and knowledge of the situation; 
Easily sorts through ambiguous cues to find ground truth; 
Reality and pilot's perception are nearly identical 
eha 


cellen 
Expert behavior Prior knowledge of similar situation used; judgment used 
o sort through conflicting data; past experiences used 
o assimilate situation; troubleshooting used that is 
beyond textbook; creative solutions used on unusual or 
ique malfunctions 


Unfavorable outcome Behavior exhibited / Substantiating criteria 


explanations 
System indicators do not corroborate actual malfunction 


Equipment malfunction 
or indicate false alarm/erroneous data 


rong decision Pilot makes incorrect choice based on cues; bad luck 


Improper procedures rong choice of procedures for situation; correct 
procedures chosen, but performed incorrectly or out of 








































Pilot does not grasp situation; reality and perception 
differ; correct decision made for wrong situation; tunnel 
ision exhibited; pilot's dominant alternative masks 
orrect one; ramifications of actions not considered 
ficient knowledge to decipher cues; ramifications 
of actions unknown; lack of systems knowledge 

0 properly troubleshoot situation 
Malfunction not covered in NATOPS; no guidance 
Available 








ck of SA 


Le 
Lack of knowledge 











Table 3-6 Explanations for Outcome 
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IV. RESULTS AND DISCUSSION 


A total of 438 hazreps covering years 1996-1999 were reviewed and analyzed (see 
Appendix B for complete list of hazreps). This chapter presents the results of the analysis 
and develops a model to illustrate the decision making process of P-3 pilots as revealed in 
this study. The chapter's first section presents the results of categorizing the decision 
events according to the class of rule-based, knowledge-based or skill-based behaviors, and 
the second section presents the results of analysis of the hazreps in terms of the major 
contributing factors for both favorable and unfavorable outcomes. The results are quite 
extensive and an enormous amount of information is presented (Appendix C contains full 
summary table). However, a majority of the results contained in the accompanying tables 
may be of more use for future studies and P-3 specific evaluations. Therefore, to maintain 
the focus on decision making, two major goals of this chapter are presented: 

-Identify those situations that are best served using an Analytical decision making 
strategy and those where a Naturalistic decision making strategy is more applicable; 

-Identify and explain the major cause of unfavorable outcomes in each of the rule- 
based, knowledge-based and skill-based classes. 

Accomplishing these goals will provide a basis for identifying requirements to 


better train and equip decision makers in an aviation environment. 


A. CLASS OF DECISION BEHAVIOR 
The hazreps were categorized according to the class of decision behavior (using 


—_ 


the characteristics of decision processes identified by Rasmussen: knowledge-based, rule- 
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based or skill-based behavior), type of aircraft system that was most responsible for the 
submission of the hazrep and the category that most accurately describes the situation 
encountered in the hazrep. Table 4-1 presents the percent of hazreps for each category 
based on the type of decision making behavior/environment. 


1. Rule-based Decisions 

Rule-based overall. The greatest number (78%) of hazreps were categorized as 
involving rule-based behaviors. This high percentage was expected and is attributable to a 
sound Safety/NATOPS program and years of observation and experience. The intent of 
the hazrep is to reduce future hazards by identifying them and finding and incorporating 
into the NATOPS publications a subsequent "cure" to prevent their recurrence. One of 
the ways the Safety/NATOPS program accomplishes this task is by providing rules, 
procedures and checklists for a crew to follow in the event of a malfunction. The 
NATOPS program is under constant review; changes to the flight manual are submitted 
continuously and a conference held annually to review these submissions for additions to 
the manual. The NATOPS manual has grown considerably over the years as new hands 
are identified (and procedures modified to best handle a situation) and is often referred to 
as a manual that is "written in blood": unfortunately, the blood of previous aviators that 
had the misfortune of not having a checklist or procedure to guide them. Ideally, the rule- 
based category would contain 100% of the hazreps and all malfunctions would have a 
corresponding procedure or textbook solution. 

The rule-based class was divided into two subclasses, based “a whether or not 


judgment was applied when deciding on a course of action. These are: rule-based simple 
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and rule-based judgment. The judgment subclass contains those hazreps where pilot 


discretion was used prior to deciding on a course of action. However, the prescribed 


e*D 0350 Qc un 


Table 4-1 Results by Class of Decision Behavior, Type of Aircraft System and Category Describing 





Ambiguous indications (single COA) 
Compound factors (procedural) 
Multiple malfunctions (procedural) 
Simple procedural 

Unique; no NATOPS guidance 
Unusual malfunction ~ 


Ambiguous indications (single COA) 
Compound factors (procedural) 
Multiple malfunctions (procedural) 
Simple procedural 

Unique; no NATOPS guidance 
Unusual malfunction 


Ambiguous indications 
Compound factors 

Multiple malfunctions 

Unique; no NATOPS guidance 
Unusual malfunction 


Improper procedures 
Situational awareness 
Survival 





the Situation 


course of action was still relatively straightforward and had a clear procedure identified to 


handle the malfunction, thus it remained in the overall rule-based class. 
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Rule-based simple. Ninety percent of hazreps contained in the rule-based 
class were in the "simple" subclass (this group encompasses 70% of the total sample). 
The majority of tasks in the aviation domain are procedural in nature. Many years and 
numerous resources have gone into developing checklists and procedures for situations 
that have occurred or can be expected to occur. Since these checklists prescribe the 
appropriate courses of action for specified situations, in many instances, the task of the 
decision maker is not to generate and evaluate a course of action, but rather to accurately 
assess the situation and pick the appropriate procedure. As mentioned previously, ideally 
all situations encountered would fall into this "simple" rule-based class and aeronautical 
decision making would be reduced to merely picking the correct checklist or procedure to 
apply for any given situation. 

Rule-based with judgment. Ten percent of the rule-based class fall in the 
rule-based with judgment subclass (8% of the total sample). This number is considerably 
smaller than the rule-based simple category and indicates that some rule-based decisions 
do require discretion or judgment, even though a procedure exists to handle the 
malfunction. Troubleshooting unknown vibrations or deciding on the timing of 
procedures in a complex scenario are examples of the types of hazreps contained in this 


subclass. 


a) Cognitive Decision Strategy Used 


~.. -Although the pilots in the rule-based situations do not have to generate and 


evaluate several courses of action before making a decision to act, they still utilize an 
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analytical strategy during their decision making process. They are required to choose the 
proper checklist or procedure from a myriad of possibilities contained in the NATOPS 
manual and must use some sort of structured or logical analysis to do so. Much of this is 
learned through training and rote memorization of stimulus-response scenarios and 
requires little in the way of judgment or experience. Almost all of the hazreps in the rule- 
based category involve situations that can be handled by use of mn analytical decision 
making strategy. Analytical decision making situations are characterized in the following 
way: | 

-A choice among clearly defined options exists; 

-Complete and reliable information is available; 

-An optimal decision is desired; 

-Decisions must be justified. 

However, the ability to correctly assess a situation and choose the proper 
procedure may use cognitive skills that are based on experience and pattern recognition, 
thus exhibiting characteristics of the naturalistic decision making process. Further, some 
situations do require judgment (10% of the rule-based class in this study) and the pilot is 
more apt to use experience and knowledge of previous situations to assist in the decision 
regarding which procedure to invoke. This use of experience is characteristic of the 
pattern matching found in the recognition-primed decision strategy (a strategy that is part 
of the NDM methodology). Decisions categorized as using a rule-based behavior cannot, 


7 jw ee 


therefore,. be characterized as relying solely on an analytical decision making strategy. 
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2. Knowledge-based Decisions 


Knowledge-based. Eighteen percent of the hazreps analyzed were categorized as 
involving knowledge-based behavior. These situations involve complex decisions where 
experience and judgment are critical. The problems are such that determination of a clear- 
cut situation assessment can be difficult or the situations may be compounded by 
ambiguous information or multiple malfunctions. Judgment is required in 100% of these 
scenarios. Pilots faced with this class of decision require vast experience and years of 
training to properly assess the situation and arrive at appropriate courses of action. 
Therefore, situation assessment becomes at least as important a step as selecting from a 
checklist or implementing a procedure. A critical component of performing situation 
assessment for these situations may be to know when a situation is unusual and does not 
have a corresponding procedure. This requires that the decision maker possess a wealth 
of knowledge concerning typical malfunctions and scenarios, and that he be able to relate 
these to the current situation. Rule-based procedures may eventually be used in these 
situations, but the process used to decide on which procedure to use requires pattern 


matching, mental simulation and intuition. 


a) Cognitive Decision Strategy Used 


Experience and judgment are critical when dealing with situations that are 
ambiguous in nature or are compounded by multiple malfunctions. As the situations 
encountered increase in complexity the decision maker relies on familiarity developed 


through-experience gained as a result of having dealt with similar types of previous events. 
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This experience base also enables the decision maker to engage in mental simulation to 
attempt to gain situational awareness and "role-play" how a particular course of action 
might play out before implementing it. Both Klein's RPD and Noble's Situation 
Assessment models describe the behaviors exhibited by the decision makers in these 
complex decision making situations. Using past experience to recognize a familiar 
situation, mental simulation and role-play of future events typify recognition-primed 
decision making and are the core of knowledge-based decision methods. Almost all of the 
situations encountered in these hazreps fit the conditions that describe recognition-primed 
decision (a strategy of the NDM methodology): 

-There is a high degree of uncertainty; 

-There is ambiguous or changing information; 

-Innovative or creative thought is required; 

-Decision makers are knowledgeable and experienced. 
It must be noted that decisions made in the knowledge-based class are under the same 
scrutiny as those in the rule-based class and the decision maker must often choose the 
optimal course of action. Based on this requirement, one could argue that Analytical 
decision making is best suited for knowledge-based situations. Actually, analytical 
behaviors are invoked in deciding on a course of action once the decision maker clarifies 
the situation and begins to evaluate possible courses of action. The author believes that 
this aspect of decision making (i.e. evaluating courses of action) is better suited to the 
structured and evaluative approach of analytical decision making. However, the use of the 


recognition-primed cognitive strategy (pattern matching, mental simulation, expert 
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situation assessment) is also critical to properly execute the process in its entirety. 
Therefore, as is the case with the rule-based class, knowledge-based situations require 
cognitive strategies from both the analytical and naturalistic methodologies. 


3. Skill-based Decisions 

Skill-based. Four percent of the hazreps analyzed were categorized as involving 
skill-based behaviors. These situations requiring skill-based behavier dealt primarily with 
preflight errors and the results associated with the improper performance of procedures. 
Since the hazreps in these instances were usually a result of previous behavior, the 
decision making process cannot be characterized due to a lack of information in the 
hazrep. The event described in the hazrep was an explanation for the handling of the 
resultant malfunction and did not address behaviors that caused the performance of 
improper procedures in the first place. Based on the experience and expertise of the 
author (often in corroboration with remarks made by the Commanding Officer in the 
hazrep) it is apparent that most of the errors were caused by a lack of situational 
awareness. The skill-based class is included in the study to illustrate the ensuing problems 
that can occur with an error in skill-based behavior, such as when a flight engineer leaves 
the oil cap unfastened which results in an oil leak inflight and a subsequent three engine 
landing. 

Also included in this class are the basic stimulus-response, or automatic reactions, 
to near mid-air collisions. No judgment is required as this is an instantaneous reaction to a 


life-threatening situation. This is an example of a learned response that is so well learned 


that the resulting behavior is almost innate to the decision maker and requires very little 
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conscious effort or thought. An error in this skill-based behavior would obviously have 


grave consequences. 


a) Cognitive Decision Strategy Used 


The limited information contained in these situations categorized as 
involving skill-based behavior makes it difficult to determine what type of decision making 
cognitive strategy is being used when preflight errors occur. ee the characteristics 
of the recognition-primed decision making strategy, the instantaneous, or intuitive, 
reaction to an approaching aircraft could be considered a case of very rapid pattern 


matching and could be considered to fit in this category. 


4, Type of Aircraft System 


Engine and propeller malfunctions were the major system contributors across the 
entire sample of 438 hazreps. This pair of system contributors was expected, as the 
engine and propeller systems are the two most complex systems on the aircraft and have 
far more moving parts and accessories than other aircraft systems. Categorizing by type 
of system contributing to the malfunction does not provide much data in the analysis of 
decision making, but may provide valuable information for engineering, manufacturing and 
rework facilities for the P-3, as it highlights two relatively high failure systems. 


5. Category Describing Situation 


a) Rule-based Decisions 


Simple procedural was the standout category leader in describing the 


situation encountered in the rule-based class, accounting for 85% of the total. As 
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mentioned previously, this is expected, as the intent of the Safety/NATOPS program is to 


provide a simple "by the book" procedure for any encountered malfunction. 


b) Knowledge-based Decisions 


Situations involving compound factors (38%) and ambiguous indications 
(33%) combined to make up 71% of the situations encountered in the knowledge-based 
class. These situations were not covered explicitly in the NATOPS and require the 
decision maker to use judgment, expertise and experience to correctly assess the situation 


and decide on a course of action. 


B. OUTCOME AND EXPLANATIONS OF HAZREP SITUATIONS 


The results included in the hazreps regarding favorable and unfavorable outcome 
were quite informative and are discussed below. Table 4-2 presents the results of the 
analysis. It appears that the likelihood of a favorable outcome decreases as the complexity 
of the problem situation increases. Clearly interpretable, unambiguous and 
uncompounded situations are more likely to produce a favorable outcome, as rules and 
procedures exist to handle them. 


1. Rule-based Decision Outcomes 


The rule-based simple class comprised the largest category and had the highest 
percentage of favorable outcomes at 92%. This was expected, as the simple rule-based 
situations merely require a quick and easy situation assessment followed by the selection 


and application of the corresponding procedure. A high level of expertise is not 


_- 
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Favorable outcome Unfavorable outcome 
% | % % % 


Equip malfunction 
Improper procedures 


Correct decision; 100 Lack of knowledge 
Expert behavior| 0 
No NATOPS guidance 
Rule Wrong decision 


78% Equip malfunction 
Improper procedures| 
Correct decision] 64 Lack of knowledge] 
Expert behavior| 36 Lack of SA 
No NATOPS guidance 

Wrong decision 





e*3 OD 3Z3Qaca 
~ 
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Equip malfunction| 29 
Improper procedures} 11 
Lack of knowledge} 11 
Lack of SA; 46 

No NATOPS guidance| 0 
Wrong decision} 3 


- Equip malfunction| 0 

Improper procedures| 29 

Skill 22 |Excellent SA | 78 Lack of knowledge| 0 
4% Lack of SA] 71 | 
| No NATOPS guidance} 0 | 

Wrong decision} 0 


Table 4-2 Results by Outcome and Explanation of Hazrep Situations Encountered 


| Correct decision 


Expert behavior 





necessarily required and the malfunction can usually be handled quickly and effectively 
through applying the clear guidance contained in the NATOPS flight manual. 

Only 8% of the rule-based category had unfavorable outcomes, and the largest 
contributor for an explanation of these unfavorable outcomes was “equipment 


malfunction" (72%). This equipment malfunction is different from an equipment failure 
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that is considered routine and expected (e.g. chips light or oil leak), and is the result of a 
failure of a detecting or indicating system (e.g. oil transmitter failure with oil system 
operating 4.0, or fire warning detection system that gives an erroneous indication). In 
fact, almost all of the pilots in these situations correctly performed procedures for the 
indications exhibited, but the outcome was still considered "unfavorable" since the 
malfunction was actually a false alarm and involved an unnecessary abort. 

The rule-based judgment class is relatively small and there were fewer favorable 
outcomes (76% favorable) than in the rule-based simple category. This is probably due to 
the increased complexity of the decisions when judgment is required. Judgment relies on 
the expertise, experience and discretion of the pilot and is affected by the situation. 

The rule-based judgment subclass involves decisions that are simpler than those in 
the knowledge class and have procedures more or less spelled out, but still have room for 
error in performing situation assessment and/or the selection of appropriate procedures. 
While most of the rule-based situations merely require a stimulus-response type behavior 
(e.g. "if 'x' occurs, execute 'y'"), there is a chance that the decision maker may misinterpret 
the stimulus "x" or fail to consider all the cues presented in the situation. The correct 
procedure in the decision maker's mind might actually be contrary to the actual procedure 
required. An example of this type of incorrect selection of the appropriate procedure is 
the pilot who mistakenly seoatis pitchlocked propeller procedures for a propeller that 
has not yet pitchlocked. In his assessment of the situation, the pilot mistakenly believes 


the propeller to have already pitchlocked and performs the procedures accordingly. 
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"Improper procedures" (50%) was the leading cause of unfavorable outcomes in 
the rule-based judgment subclass. It appears that incorrect situation assessment was the 
major basis for the performance of improper procedures and can possibly be explained by 
one of the following. The decision maker incorrectly assessed the situation and selected 
procedures based on this incorrect assessment (pitchlocked propeller case above); he 
assessed the situation correctly but performed the wrong procedure; Or he was locked into 
a dominant alternative and failed to reassess a changing situation, thus performed the 
wrong procedure. 


2. Knowledge-based Decision Outcomes 

Ambiguous indications (33%) and compound factors (38%) combined accounted 
for 71% of the hazreps (as shown in Table 4-1) in the knowledge-based class. The 
percentage of favorable outcomes was 65% and was due largely to expert behavior (73% 
of the favorable outcomes were attributed to this). Commanding Officer's comments on 
favorable hazreps all referred to the experience, sound judgment and professionalism 
exhibited by the pilots in the favorable outcome situations. 

Unfavorable outcomes (35%) were due primarily to a lack of situational awareness 
(46% attributed to lack of SA). This clearly illustrates the need for experience and 
expertise in the cockpit. Many of the hazreps with unfavorable outcomes contained 
Commanding Officer's comments regarding the inexperience of the crew or lack of sound 
decision making skills. "Equipment malfunction" (29%) was the second leading cause of 
aii tans and it can be easily seen that erroneous ‘otications (discussed 


previously under rule-based simple class) can complicate an already ambiguous situation. 
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Again, this was a system indicating problem and does not imply wrong-doing on the part 


of the pilot. 

3. Skill-based Decision Outcomes 

The favorable outcomes (22%) were all directly attributable to excellent SA on the 
part of the pilot in avoiding a mid-air collision. The 78% of unfavorable outcomes were 
attributed primarily to a lack of SA on the person responsible for the preflight error that 
caused the hazrep situation (refer to Class of Decision Making Behavior 'Skill-based' 


earlier in the chapter). Table 4-3 summarizes the results discussed above. 


#1 % | #1 
% Favorable | Explanation| Unfavorable | Explanation 
Simple decision 
Judgment 16 decision 24 procedures 
Mowrased| 65 [oohavior | 38 (uackof SA 
based behavior | Lack of SA 


Table 4-3 Outcome synopsis of explanations for favorable and unfavorable situation outcomes 













C. MODELLING DECISION MAKING 


A simplified model of decision making based on the analysis conducted in this 
study is presented in Figure 4-1. It illustrates the inputs and outputs to memory as the 
processes of the OODA loop discussed in Chapter II. As can be seen, a decision maker 
attempts to gain situational awareness by first processing the observed cues in working 


memory.” If an immediate response is not readily apparent, the decision maker must 
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attempt a pattern match with templates of prior experience stored in long term memory. 
A decision is then derived in working memory and an appropriate action performed. 

Figure 4-2 is a more elaborated model of the decision processes used by pilots for 
the problem situations encountered in this study. It is a hybrid model that combines 
properties of the OODA loop, cognitive hierarchy, pattern matching model and 
Rasmussen's (1983) decision behaviors. The figure shows two major processes working 
concurrently: the decision making processes of the OODA loop (Observe, Orient, Decide 
and Act) and the information processes of the cognitive hierarchy (data being transformed 
to information, and then to knowledge, and finally to understanding). The boxes (outlined 
in bold) of the OODA loop represent major stages in the processing of information, two of 


which are located in working memory. Although working memory is depicted as two 


separate levels, it should be thought of as one entity, as depicted in Figure 4-1. 


S 
RKC Long Term 
ONS Memory 
ee 





— 


Figure 4-1 Simplified decision model depicting inputs and outputs to memory 
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As can be seen, skill-based and rule-based simple decisions require little effort from 
working memory. As the pilot processes the information and begins the orient phase, he 
either reacts immediately to the stimulus (e.g. near mid-air collision avoidance) or readily 
pattern matches the cue to a specific procedure or checklist. Errors resulting in 
unfavorable outcomes at this stage are caused primarily from erroneous systems 
indications and not human error. Extensive training and repetition of procedures produces 
pilots that are quite effective at matching the procedure to the nainineion and then 
performing the procedures correctly. Only when a pattern match is not obtained 
immediately does the decision maker have to process the information further and resort to 
long-term memory for a possible stored event that is congruent with the situation at hand. 

This venture into long-term memory uses cognition and learned aspects of the 
domain (e. g., failure modes, operating characteristics, previous hazreps, etc.) and results 
in the pilot gaining a clearer situation assessment or "knowledge picture" (refer to 
cognitive hierarchy discussion in Chapter II) of the situation. The author believes that the 
more effectively the decision maker can perform pattern matching in working memory 
(after retrieving an experience from long-term memory), the more effectively the problem 
situation can be handled. This process is representative of the expert behavior seen in the 
knowledge-based hazreps and 1s best performed by more experienced pilots. Experienced 
pilots have a much broader knowledge base to draw from and can gain situational | 


awareness faster. 
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Figure 4-2 Hybrid decision model for P-3 pilot hazrep decision making in problem situations 





Once situational awareness is obtained the pilot must decide on a course of action. 








This process is once again centered in working memory and can heavily tax the decision 


maker's attentional resources and reasoning ability. The rule-based judgment class is less 


demanding on working memory and does not require the greater understanding that 


knowledge-based decisions do. If previous knowledge, past experience or domain specific 


knowledge have not provided a pattern match (or similar match) then the pilot must 
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integrate cues and generate an original solution to the problem. This creative process may 
further tax the attentional resources of working memory. 

The "loss of SA" cloud shown in the upper left corner of the figure expands as 
stress and time-pressure increase, effectively reducing the capability of working memory. 
Losing situational awareness or performing improper procedures for the malfunction at 
hand most often causes errors in this region. Again, experience may be the critical 
component to alleviating this loss of SA as uncertainty and stress increase. 

A caution regarding experience must be mentioned at this point. It is conceivable 
that a pilot with extensive past experience could "fixate" on an alternative that is not 
appropriate for the situation. As working memory capacity is taxed and the SA cloud 
grows, one alternative may dominate the decision maker's judgment so that it is seen as 
the only viable alternative. Under stress (e.g. ambiguous information, time-pressure, etc.) 
decision makers are prone to develop a type of "cognitive tunnel vision." 

Not explicitly shown in the figure is a feedback loop that occurs after an action has 
been performed. This feedback presents itself as data entering the observation phase. 


Thus, the cycle is an iterative loop that allows for flexibility and change. 
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Vv. CONCLUSIONS AND RECOMMENDATIONS 


This thesis examined decision making in a P-3 flight station during malfunction 
situations. An analysis of 438 hazreps was conducted to distinguish different kinds of 
decision situations encountered and whether a particular decision strategy from either the 
Analytical or Naturalistic decision making methodologies is more appropriate for a given 
situation. Situation outcomes were analyzed to pinpoint the cause of unfavorable results 
in an attempt to prevent their reoccurrence in the future. 

It is clear from the results that the majority of situations analyzed involve rule- 
based decisions and can be performed with relative ease by matching a procedure to a 
malfunction. iiaes complex decisions are required when the situation is clouded with 
ambiguity or compounded by additional factors. These knowledge-based decisions 
require judgment and experience on the part of the decision maker. Experience appears to 
be the critical component in gaining and maintaining situational awareness and making 
effective decisions. Strategies from both the Naturalistic and Analytical decision making 
methodologies have a place in the decision making processes of pilots in this study and 
must be considered when training new personnel. Table 5-1 summarizes the key aspects 
of the study and provides the following observations: 

1) Rule-based decisions comprise the vast majority of situations encountered in 

the aircraft; 

2) Knowledge-based decisions require experience, intuition, previous knowledge 


and expert behavior: 
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3) The probability of an unfavorable outcome increases as the complexity of the 
environment increases; 

4) A need exists for better detecting and indicating systems for malfunctions; 

5) Situational awareness is key to preventing unfavorable outcomes; 


6) Recognition-primed and analytical decision making strategies are both required 


— i 


for effective decisions. 


Viain decision making i) | 
% method Unfavorable | Explanation 
Equipment 
Rule simple 70 {Analytical malfunction 
Rule judgment Analytical w/ NDM procedures 
Knowledge Naturalistic w/ ADM | Lack of SA 
Skill Naturalistic (NMAC) Lack of SA 


Table 5-1 Type of decision making strategy associated with class of decision type and explanations 
for unfavorable outcome 












A. RECOMMENDATIONS 


Recognizing the need for the use of strategies from both the Analytical and 
Naturalistic Decision Making methodologies is critical if the Navy wants to reduce the 
number of unfavorable outcomes in hazardous situations encountered in Naval Aviation. 
The recommendations below are intended to: 

- Increase situational awareness among pilots in order to reduce the probability of 
errors associated with improper situation assessment; 


- Alleviate erroneous aircraft indicating systems in order to reduce ambiguous 


—_ 


situations and unnecessary aborts. 
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1. Train Pilots to Become Experts 


The military conducts ab initio training; that is, it transforms complete novices into 
minimally proficient and safe pilots (Kaempf & Orasanu, 1997). The basic flight training 
takes approximately one year, followed by specific in-model training at a Fleet 
Replacement Squadron and an upgrade syllabus in an operational squadron. A typical P-3 
Plane Commander requires approximately 18-24 months to completethe upgrade syllabus. 
After designation, a fully upgraded pilot has little in the way of training requirements other 
than annual refresher training. This type of training produces a vast number of competent 
pilots that have very little experience to draw from. In addition, a pilot's tour (especially 
Navy pilots) not only involves flying but also performance of a ground job and associated 
military duties and watches. Flying frequently becomes a secondary task. The result is a 
pilot who is proficient at handling the routine rule-based decisions but does not necessarily 
have the expertise to handle the more complex knowledge-based decisions. 

The first prerequisite for improving situational awareness and the skills to perform 
the pattern-recognition strategy is to increase expertise. The more comprehensive a pilot's 
knowledge, the easier it is to find a pattern match in long term memory to deal with a 
hazardous situation. Pilots should practice decision making skills in time-limited and 
stressful situations on a daily basis. Training must not be confined to staff courses, annual 
refresher training or postgraduate education, but be instituted on a regular basis in 


operational squadrons. Decision training should be incorporated into flight training from 


bee. YET 


the onset and continued throughout a pilot's career. 


_ 
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Personal experience is the best way to acquire expertise, but this is not always 
possible. The use of simulators, decision making exercises and “hangar flying" must be 
mandated. Simulation provides an excellent method of training on a large number of 
challenging and realistic scenarios in a short period of time. Situations can be 
programmed in simulators to replicate actual aviation hazards that have occurred in the 
past. Simulation also can present the decision maker with the problem of analyzing 
ambiguous or complex situations without the fear of failure (e.g., loss of life or aircraft). 
Using historical analyses of hazardous situations and real time feedback can vastly 
improve the decision making processes of pilots. 

This is not to say that simulators are not currently used in the operational fleet; in 
fact, they are an integral part of all the training tracks that the author has been associated 
with. However, once a pilot completes his/her prescribed training and becomes a Plane 
Commander; there is a dramatic decrease in mandatory simulator training required. The 
time that is required in simulation is usually spent conducting instrument proficiency 
training rather than improving decision making skills. 

Group decision making exercises and pilot forums also provide an excellent means 
of training junior (qualified and unqualified) pilots. Reviewing orevious hazardous 
situations and discussing the way in which the they were handled by other decision makers 
can build a pilot's experience base. Further, a group discussion evokes other's opinions on 
alternative courses of action and the decision strategy used to arrive at that course of 


™~ 


action. These opportunities provide an excellent resource for transferring experience and 


expertise directly from one pilot to another. 
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2. Utilize Existing Expertise 

A Naval aviator frequently is given an enormous amount of responsibility while 
still very junior. For example, the author was a P-3 Plane Commander as a LTJG and a 
Mission Commander and Instructor Pilot as a junior LT. While this amount of 
responsibility is sought after and a tribute to one's abilities, it may not be in the best 
interest of properly training junior decision makers. A better approach may be to pair 
novice aviators with seasoned and Sqenenced pilots for a longer period of time after the 
novice is qualified. This gives the newly designated Plane Commander the opportunity to 
hone the decision making skills necessary for making the more complex knowledge-based 
decisions. Furthermore, this also provides the experienced pilot a fresh "rule-based 
philosophy” from the junior decision maker that is not clouded by bias, prejudice or ego. 
This naive, "book-learned" perspective from the junior aviator could alleviate unfavorable 
outcomes in hazardous situations caused by experienced pilots who persist in 
implementing undesirable (dominated) alternatives when better choices are available. 


3. Incorporate Naturalistic Decision Making Strategies in the Training of 
Pilots 


This study exemplifies the need for training decision makers in strategies from both 
the Analytical and Naturalistic Decision Making methodologies. However, the application 
of NDM research in aviation is still in its infancy. To the best of the author's knowledge, 
there are no courses presently taught in the P-3 aviation community that address NDM. 
Naturalistic. decision making strategies are most applicable when encountering dynamic 


and continually changing conditions, real time reactions to these changes, ill defined tasks, 
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time pressure, significant personal consequences for mistakes, and eepenenced decision 
makers. As this study illustrates, these conditions prevailed in at Jeast 18% of hazardous 
situations; i.e., when knowledge-based decisions were required. Using the recognition- 
primed strategy, experienced decision makers assessed the situation and recognized 
familiar patterns. In fact, the recognition-primed strategy was probably used in virtually 
every situation encountered for the initial situation assessment. It is vital that pilots be 
aware and trained in a decision making methodology that may help them make a life 
saving decision in the future. 


4. Improve Reporting Procedures to Include Decision Making Strategies 
Utilized 


While conducting this analysis of P-3 hazreps the author discovered that the hazrep 
message was lacking in information that would have proved helpful in deciphering the 
sesaioh making process used. The basis for the pilot's situation assessment (e.g., 
NATOPS, id experience, inferred from historical knowledge, etc.), discussions 
regarding how a decision was arrived at, alternatives considered, and whether time 
constraints played a vital role in the decision process could have shed more light on the 
subject. A "decision strategy" section of the hazrep could provide pilots a valuable 
discussion item for their group decision making exercises and "hangar flying" sessions. 
The practice of providing complete and forthright information in the hazrep must not only 
be encouraged, but enforced, to ensure that all the actions of the decision maker, right or 
wrong, are-clearly explained and documented. In this way, junior decision makers can 


evaluate the situations and actions for themselves and build their own experience base. 
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5. Utilize a Decision Support System 


Decision support systems can be utilized to provide the decision maker with timely 
and accurate information to aid the decision making process. Recall that the largest 
contributor for an explanation of unfavorable outcomes in the rule-based simple class was 
"equipment malfunction" (72%). This equipment malfunction is different from an 
equipment failure that is considered routine and expected (e.g. chips light or oil leak) and 
is the result of a failure of a detecting or indicating system. A decision support system 
could not only provide valuable insight to the decision maker to prevent these unnecessary 
aborts, but also could aid the pilot with situation assessment. 

One such system currently under development by the USAF aa NASA is Hazard 
Monitor (HM). Hazard Monitor is a knowledge-based aid designed to reduce the rate of 
preventable accidents regardless of the source of the problem (i.e., human, machine, or 
external environment). The idea for the system stems from the fact that in complex 
situations, hazards occur despite improvements in system design and advances in human- 
computer interaction. HM aids these situations by enhancing the problem recognition and 
identification process so that operators recognize deteriorating situations in time to avoid 
adverse consequences. (Ernst-Fortin et al., 1997) 


6. Replace Faulty Aircraft Indicating Systems 


In the absence of a decision support system, simply modifying the aircraft to 


reduce the number of false indications would greatly reduce the number of hazardous 


* 2 


situations encountered. Faulty fire warning elements and oil pressure/quantity transmitters 


- . 


were the high failure items and accounted for a majority of the "equipment malfunction" 
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category. Failures involving fire warnings and/or the oil system usually result in a three 
engine landing. While three engine landings are well trained for and often considered 
"routine" almost all Commanding Officers will tell you "four engines are always better 
than three." The cost of replacing faulty indicating systems would most likely be far 


outweighed by the benefits realized. 
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APPENDIX A. SAMPLE HAZREP ANALYSES 


Appendix A contains selected sections from three sample hazreps from each of the 
classes (knowledge-based, rule-based and skill-based) with an accompanying analysis 


based on the rules and criteria outlined in Chapter IIT. 
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KNOWLEDGE-BASED DECISION EXAMPLE 


EVIDENCE AND ANALYSIS. Immediately following takeoff at NAS 
JAX for a USW event, #1 chips light illuminated with no secondaries. PPC 
requested downwind, and once established, shutdown #1. The restart 
checklist was completed through PCO. The following items were then 
discussed: restarting #1, dumping the remaining fuel in tank 5, weather, 
and overweight landing. Two engine performance based on gross weight 
and OAT was determined. Of note, landing ground roll distance was not 
discussed. The PPC elected to perform an overweight landing based on the 
two engine performance and the forecast of Thunder I conditions prior to 
arriving at 114,000 lbs. The three engine landing was performed on 
runway 27 (dry, 8000 ft) at approximately 120,000 Ibs, with 2000 Ibs of 
fuel remaining in tank 5. Winds were 180/6. Landing touchdown speed 
was 142 kts with 6500 ft remaining. Brakes were not utilized until 
approximately 2500 ft remaining. The aircraft was stopped with 
approximately 200 ft remaining. Brake cooling was neither discussed nor 
performed. Taxiing into the parking spot and following engine shutdown, 
the lineman signaled hot brakes on port side followed by a brake fire signal. 
Crew called for assistance and egressed over starboard wing. 


REMARKS.  Postflight discussions about the decision to land heavy 
brought up items for concern. 1. Post flight determination of the landing 
ground roll distance was computed to be about 4200 ft with moderate 
braking. This figure probably would not have prevented the PPC from 
making the overweight landing but it should have spotlighted the need for 
brakes and perhaps brake cooling. 2. The PPC used the absolute worst 
case scenario to arrive at his decision to land the aircraft at 120,000 Ibs; the 
potential loss of another engine and forecast convective weather. This 
decision path reduced his options and shortened his timeline. 


CO COMMENTS. The PPC decided that the potential for another engine 
malfunction and worsening weather was more hazardous than the risks 
associated with an overweight three engine landing on a hot Jacksonville 
day. The crew discussed what could happen if they remained 1n the air but 
they failed to fully discuss what could happen if they landed in their present 
configuration. Overlooked were items including ground roll distance, 
moderate braking and brake cooling procedures. Although an "uneventful" 
landing ensued, the brake fire/hot brakes were only one of the many 
-possible outcomes of this scenario. The more prudent course of action 
would have been to continue overhead the field, prepared to relight the 
feathered engine and burn down to a lower weight prior to landing. 
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ANALYSIS OF KNOWLEDGE-BASED HAZREP 


Type: Engine (chips light) 


Class: Knowledge (multiple alternatives available, thorough understanding of 





environment required before taking action) 

Category: Compound factors (heavy aircraft, weather) 

Judgment _required: Yes (PPC elected to land heavy, PPC chose worst case 
scenario) 

Favorable outcome: No (resulting brake fire could have been avoided if better 
decision made) 

Explanation: Lack of SA (PPC perceived his worst case scenario as the deciding 
factor; crew failed to fully consider ramifications of landing heavy) 

Comments: The question may arise, if the outcome had been favorable, would the 
explanation have been "expert behavior?" The answer is no. If the PPC had elected to 
wait overhead the field and commence the approach at a lighter gross weight, or if he had 
considered the use of brakes and brake cooling procedures, the explanation would have 
been "correct decision." Full consideration of performance factors and proper application 
of NATOPS procedures is not considered "expert" behavior. Expert behavior requires 
going beyond normal procedures and uses experience, judgment and (sometimes) prior 


knowledge to arrive at a decision. 
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RULE-BASED DECISION EXAMPLE 


EVIDENCE AND ANALYSIS. On station at 2550 ft AGL, crew had two 
engines loitered at a gross weight of 100,000 Ibs. Crew thoroughly 
discussed three and two engine loiter considerations and determined single 
engine rate of descent to be 420 FPM. Number one engine had been shut 
down for 5.5 hours and #4 engine for one hour when FE called out #3 
RPM flux with decreasing RPM toward zero percent with no audible 
indications. With PPC concurrence, FE initiated restart on #4 engine. 
During-restart, FE called #3 oil pressure low light, master electrical power 
light and master pressurization light. As FE called #4 normal restart, #3 
chips light illuminated. Crew secured #3 engine and completed emergency 
shutdown checklist through item 6. With power on #4 limited due to low 
oil temp, the PPC initiated a 500 FPM rate of descent to maintain loiter 
airspeed. Engine #1 was restarted with normal indications. Crew 
completed emergency shutdown checklist for #3 engine followed by restart 
checklist for engines #1 and #4. Power became available on engine #4 
within one minute of restart and within three minutes for engine #1. 
Aircraft minimum altitude 1800 AGL. Crew initiated climbout for RTB 
and checked offstation while discussing considerations for a second engine 
failure. Weather was updated, crew briefed three engine landing 
considerations, declared an emergency, and conducted an uneventful three 
engine landing. 


REMARKS. Postflight inspection revealed a cracked idler gear in #3 
reduction gearbox which resulted in loss of entire RGB accessory section. 


CO COMMENTS. Solid crew coordination is required to safely conduct 
two engine loiter operation. Initial engine #3 malfunction indications did 
not require immediate action, so the PPC made the decision to restart #4 
first. By the time additional indications required the shutdown of #3 
engine, #4 was on line. Although #4 was available if needed, with 
sufficient altitude to accept a descent while waiting for engine oil 
temperature to increase, the PPC restarted #1. Total altitude lost was 700 
ft, suggesting both engines were on line in less than 2 minutes. Clear 
understanding of the two engine loiter brief, sound decision making and 
" professional execution ensured safe recovery from a situational risk a PPC 
accepts when he decides to loiter two engines. 
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ANALYSIS OF RULE-BASED HAZREP 

Type: Engine (RGB, chips light) 

Class: Rule-based (NATOPS dictates action) 

Category: Compound factors (two engine loiter, altitude and rate of descent 
considerations) | 

Judgment required: Yes (PPC elected to restart #4 prior to securing #3, held 
emergency shutdown checklist prior to completion to perform restart on #1) 

Favorable outcome: Yes (correct decisions made and actions taken fit the 
situation) 

Explanation: Expert behavior (PPC fully aware of status of #3 engine, judgment 
used to delay securing of #3 engine with indications of pending failure, prioritization of 
procedures required to minimize altitude loss and acceptance of slight loss of altitude to 
maintain loiter airspeed) 

Comments: This particular emergency situation could have easily ended in disaster 
for another crew faced with the same situation. A misinterpretation of the indications on 
#3 could have led a crew to securing it prior to initiating restart on #4. The aircraft would 
have then been operating on a single engine with a rate of descent of 420 FPM. Further, if 
a PPC disregarded the need to maintain loiter airspeed during the restarts, he could easily 


place the aircraft in a stall as airspeed bleeds off. 


~ . 
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SKILL-BASED DECISION EXAMPLE 


EVIDENCE AND ANALYSIS. The crew had just completed starting the 
#2 engine utilizing a huffer in preparation for a functional check flight 
(FCF). After making a quick scan of the instruments and noting a normal 
start, both the PPC and the CP noticed the aircraft was slowly moving 
forward. Both pilots quickly depressed the brakes, which immediately 
stopped the aircraft after an estimated six feet of movement. 
Simultaneously the lineman chocked the main landing gear, ensuring no 
further movement of the aircraft was possible. Neither pilot noticed if the 
brake handle was set prior to using the brakes to stop the aircraft. The 
aircraft was secured and maintenance informed. The entire brake system 
was checked with no discrepancies found. 


REMARKS. After reviewing the scenario, the PPC decided that he must 
not have ensured the brakes were properly set during the before start 
checklist. With no specific recollection of any deviation from standard 
procedure, the only explanation can be inattention during the checklist. 


CO COMMENTS. This highly qualified crew was faced with the challenge 
of a short notice FCF. Such flights occur in the course of normal 
operations and are a part of doing business. The learning point is the need 


for schedulers and crews to identify this type of added hazard in their ORM 
process. Heightened attention to detail could have prevented this scenario. 


ANALYSIS OF SKILL-BASED HAZREP 
Type: Other (basic skill error) 
Class: Skill-based (procedure exists on "Before Start" checklist to set parking 
brake---a common skill) 
Category: Improper procedures (did not perform checklist 1tem) 
Judgment required: No 
Favorable outcome: No 
| Explanation: Improper procedures (see category) 


Comments: None 


76 











APPENDIX B. LIST OF HAZREPS 
Appendix B contains the complete list of hazreps (438) analyzed in this study 


(Excel format). The hazreps are sorted by class (knowledge-based, rule-based and skill- 
based) then by category that best describes the situation (ambiguous situation, compound 


malfunction, etc.). 
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APPENDIX C. SUMMARY OF RESULTS 
Appendix C contains a summary table of the results presented in Chapter IV. The 


table is only a summary of the three tables contained in Chapter IV and has no new or 


additional information. 
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